Papers with fine-tuning technique
Trigger Word Detection and Thematic Role Identification via BERT and Multitask Learning (D19-57)
Copied to clipboard
| Challenge: | Using natural language processing to discover and mine drug-related knowledge from text has been a hot topic in recent years. |
| Approach: | They propose to use a pre-trained biomedical language representation model to extract mutation-disease knowledge from PubMed. |
| Outcome: | The proposed approaches achieve 0.60 (ranks 1) and 0.25 (rank 2) on task 1 and task 2 respectively in terms of F1 metric. |
HFT-CNN: Learning Hierarchical Category Structure for Multi-label Short Text Categorization (D18-1)
Copied to clipboard
| Challenge: | Existing methods for categorization of short texts use non-hierarchical flat model, but they are limited by domain-independent knowledge distribution. |
| Approach: | They propose a method which leverages hierarchical relationships between pre-defined categories to tackle the data sparsity problem. |
| Outcome: | The proposed method is competitive with the state-of-the-art methods on a multi-label categorization task for short texts using two benchmark datasets. |
HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing techniques to fine-tune pre-trained language models on downstream tasks are inadequate. |
| Approach: | They propose a technique to perturb hidden Transformers representations by enhancing generalization of hidden representations from different layers. |
| Outcome: | The proposed technique outperforms vanilla fine-tuning and enhances generalization of hidden representations from different layers. |
RoAST: Robustifying Language Models via Adversarial Perturbation with Selective Training (2023.findings-emnlp)
Copied to clipboard
Jaehyung Kim, Yuning Mao, Rui Hou, Hanchao Yu, Davis Liang, Pascale Fung, Qifan Wang, Fuli Feng, Lifu Huang, Madian Khabsa
| Challenge: | Several perspectives of robustness for pre-trained language models have been studied independently, but lacking a unified consideration in multiple perspectives. |
| Approach: | They propose a technique to enhance the multi-perspective robustness of LMs by introducing adversarial perturbation while the model parameters are selectively updated upon their relative importance. |
| Outcome: | The proposed technique improves the robustness of LMs by incorporating four perspectives on model robustness. |
Large Language Model for Multi-Domain Translation: Benchmarking and Domain CoT Fine-tuning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Achieving consistent high-quality machine translation across diverse domains remains a challenge due to limited and imbalanced parallel training data available in various domains. |
| Approach: | They propose a domain Chain of Thought technique that uses the multi-domain intelligence of LLMs to improve translation performance. |
| Outcome: | The proposed method achieves significant improvements in translation accuracy and domain robustness over traditional fine-tuning on a small dataset of four domains. |
Correcting Negative Bias in Large Language Models through Negative Attention Score Alignment (2025.naacl-long)
Copied to clipboard
Sangwon Yu, Jongyoon Song, Bongkyu Hwang, Hoyoung Kang, Sooah Cho, Junhwa Choi, Seongho Joe, Taehee Lee, Youngjune Gwon, Sungroh Yoon
| Challenge: | Experimental results show that large language models exhibit a negative bias in binary decision tasks . hallucination is a factor that degrades reliability of LLMs . |
| Approach: | They propose a negative attention score to systematically and quantitatively formulate negative bias by using a parameter-efficient fine-tuning technique. |
| Outcome: | The proposed method reduces the gap between precision and recall caused by negative bias while preserving generalization abilities. |
Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent pretrained language models extend from millions to billions of parameters. |
| Approach: | They propose a technique which forwards on a whole network while backwarding on resetting the gradients of the non-child network during the backward process. |
| Outcome: | The proposed technique outperforms the vanilla fine-tuning technique on various downstream tasks and can achieve better generalization performance by large margins. |